Deep Networks with Internal Selective Attention through Feedback Connections
نویسندگان
چکیده
Traditional convolutional neural networks (CNN) are stationary and feedforward. They neither change their parameters during evaluation nor use feedback from higher to lower layers. Real brains, however, do. So does our Deep Attention Selective Network (dasNet) architecture. DasNet’s feedback structure can dynamically alter its convolutional filter sensitivities during classification. It harnesses the power of sequential processing to improve classification performance, by allowing the network to iteratively focus its internal attention on some of its convolutional filters. Feedback is trained through direct policy search in a huge million-dimensional parameter space, through scalable natural evolution strategies (SNES). On the CIFAR-10 and CIFAR-100 datasets, dasNet outperforms the previous state-of-the-art model on unaugmented datasets.
منابع مشابه
Role of STDP in regulation of neural timing networks in human: a simulation study
Many physiological events require an accurate timing signal, usually generated by neural networks called central pattern generators (CPGs). On the other hand, properties of neurons and neural networks (e.g. time constants of neurons and weights of network connections) alter with time, resulting in gradual changes in timing of such networks. Recently, a synaptic weight adjustment mechanism has b...
متن کاملTransfer entropy-based feedback improves performance in artificial neural networks
The structure of the majority of modern deep neural networks is characterized by unidirectional feed-forward connectivity across a very large number of layers. By contrast, the architecture of the cortex of vertebrates contains fewer hierarchical levels but many recurrent and feedback connections. Here we show that a small, few-layer artificial neural network that employs feedback will reach to...
متن کاملRole of STDP in regulation of neural timing networks in human: a simulation study
Many physiological events require an accurate timing signal, usually generated by neural networks called central pattern generators (CPGs). On the other hand, properties of neurons and neural networks (e.g. time constants of neurons and weights of network connections) alter with time, resulting in gradual changes in timing of such networks. Recently, a synaptic weight adjustment mechanism has b...
متن کاملDeep Predictive Coding Network for Object Recognition
Inspired by predictive coding in neuroscience, we designed a bi-directional and recurrent neural net, namely deep predictive coding networks (PCN). It uses convolutional layers in both feedforward and feedback networks, and recurrent connections within each layer. Feedback connections from a higher layer carry the prediction of its lower-layer representation; feedforward connections carry the p...
متن کاملDirect Feedback Alignment Provides Learning in Deep Neural Networks
Artificial neural networks are most commonly trained with the back-propagation algorithm, where the gradient for learning is provided by back-propagating the error, layer by layer, from the output layer to the hidden layers. A recently discovered method called feedback-alignment shows that the weights used for propagating the error backward don’t have to be symmetric with the weights used for p...
متن کامل